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ABSTRACT 



This report contains the results of an evaluation of 
coverage issues regarding the 1994 Federal Libraries and Information Centers 
Survey. The objectives of this evaluation were to determine the accuracy and 
completeness of the final survey frame and to analyze the criteria used to 
classify federal library facilities as in-scope and out-of -scope . The report 
also points our areas for potential improvement in data collection and 
processing. The 1994 Federal Libraries and Information Centers Survey (FLICS) 
was the first statistical measurement of this group of library facilities 
since 1978. The first phase was to create a universe of all possible survey 
candidates. Nearly two-thirds of the original universe file was ruled 
out-of -scope, and over 38% of these were purposely excluded from the final 
survey frame. Comparing other library directories to the FLICS universe 
resulted in a match rate of nearly 90%. Problems were observed in the 
definition of "federal" library. Overall, the lack of alternative sources and 
corroborative data means that completeness and accuracy of survey responses, 
and therefore of the final survey frame, cannot be determined. 

Recommendations are made to alleviate these problems, re-examining the 
■federal universe. Six appendixes contains lists of different categories of 
libraries considered for the survey frame. The survey form is included. 
(Contains 15 tables.) ( SLD) 
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FOREWORD 



This report was prepared for the National Center for Education Statistics (NCES) by the Governments Division of the 
U S. Bureau of the Census, John Curry, principal author. It is intended to address issues of interest and concern to the 
NCES and education policymakers and researchers. The report contains the results of an evaluation of coverage issues 
regarding the 1994 Federal Libraries and Information Centers Survey. Its objectives are to determine the accuracy and 
completeness of the fmal survey frame and to analyze the criteria used to classify federal library facilities as in-scope 
and out-of-scope. This report also points out areas for potential improvement in data collection and processing. 
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ABBREVIATIONS USED IN THIS REPORT 



The following abbreviations are used frequently throughout this report: 

BLA — Bureau of Indian Affairs, U.S. Department of the Interior. 

DOD — U.S. Department of Defense. 

FEDLINK — Federal Library and Information Network — Library of Congress program that provides federal 
libraries, information centers, and other federal activities with information retrieval and library support services. 

FLICS — Federal Libraries and Information Centers Survey — the focus of this evaluation. 

LMC — Library Media Center Survey — the sample surveys of public, private, and Indian schools for library media 
centers. 

NCES — National Center for Education Statistics — the federal agency, within the Department of Education, that is 
responsible for collecting library statistics on a national scale. 

NEC — not elsewhere classified. 

No. — number of. 

SASS — Schools and Staffing Survey — the sample surveys of public (school and local education agency level), private, 
and Indian schools for staffing, programs, and policies. 

z — entry would amount to less than half the unit of measurement shown. 
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CHAPTER!. INTRODUCTION 

Section 1.0 Survey Background and Purpose of 
Evaluation 

The 1 994 Federal Libraries and Information Centers 
Survey (FLICS) was the first statistical measurement of 
this group of library facilities since a data collection 
effort conducted in 1978. It was also the sixth survey 
in the series and the first to cover information centers 
as well as libraries. The 1994 FLICS was conducted by 
the National Center for Education Statistics (NCES) of 
the U.S. Department of Education with additional 
support from the Federal Library and Information 
Center Committee of the Library of Congress. The 
Bureau of the Census acted as collecting agent for the 
NCES. 

A statistical survey conducted on a regular or periodic 
schedule benefits from an established universe or 
survey frame. Subsequent surveys require the panel to 
be updated with deletions and additions. But because 
so much time had elapsed since the last survey of 
federal libraries, one of the first tasks facing the FLICS 
staff was building a survey frame. 

The FLICS staff constructed a preliminary universe 
from existing directories of federal libraries, most 
notably Oryx Press’ 1993 Directory of Federal 
Libraries and the Federal Library and Information 
Network (FEDLINK) mailing list. Appendix D 
provides a complete listing of the sources that were 
used. To narrow the universe to those facilities that met 
the survey’s four-part criteria for inclusion, 1 two 
pretests and a special locator survey were conducted. 
In addition, certain types of libraries were purposely 
excluded from the final survey frame, primarily those 
located overseas or connected to elementary-secondary 
schools operated by the Department of Defense (DOD) 
and the Interior Department’s Bureau of Indian Affairs 
(BIA). 



facilities included in the survey were: (1) either a library 
or an information center, (2) staffed with at least one paid 
part-time or full-time librarian , technical information 
specialist, library technician, archivist, or other trained 
person whose principal function was to assist others in 
meeting their information needs, (3) considered to be a 
federal government operation or receiving a majority (at 
least half) of its funding from federal appropriations, and (4) 
in support of the information needs of a federal agency or 
supplied information as part of the agency ‘s mission. 



The primary focus of this evaluation was on the 
accuracy and completeness of this newly-created 
survey frame. A secondary objective was to analyze 
the survey’s criteria for classifying facilities as in-scope 
or out-of-scope. Specifically, the evaluation looked 
into four major coverage issues: 

1. Nearly two-thirds of the library facilities on the 
original universe file failed to meet the FLICS ’s 
criteria for inclusion in the final survey frame, a 
percentage that seemed high. Chapter 2 discusses 
and presents selected statistical tabulations 
comparing the characteristics of the in-scope and 
out-of-scope groups. 

2. Twenty-four percent of the universe was purposely 
excluded from the final survey. Chapter 3 
explores the validity of the assumption underlying 
the largest group of excluded federal libraries, 
namely that the libraries connected to DOD and 
BIA-operated schools were canvassed in other 
NCES-sponsored library surveys. Appendix A 
provides a complete listing of the 822 purposely- 
excluded facilities. 

3. The FLICS staff employed the major directories of 
federal libraries to construct its original universe. 
Available for this evaluation were secondary 
sources listing federal libraries that were compared 
to both the universe and final survey frame. This 
comparison, the topic of Chapter 4, resulted in the 
identification of 189 library facilities that could 
not be matched to the FLICS’ files. These 
unmatched facilities are listed in Appendix B. In 
addition, this comparison revealed 312 facilities 
that were considered to be libraries in one or more 
secondary source and which were on the FLICS 
universe, but failed to qualify for the final survey 
frame. This subtopic is also discussed in Chapter 
4 while the 312 facilities are listed in Appendix C. 

4. The final phase of this evaluation looked at the 
FLICS’ definition of “federal” as applied to the 
library universe. As discussed in Chapter 5, the 
survey applied a two-part defmition of what 
constitutes a “federal” library. The first part 
specified agencies that were “considered” to be 
federal, but left that decision to the respondent 
rather than basing it on established listings. The 
second part specified libraries whose funding was 
paid primarily from federal funds (which met the 
other three criteria). Chapter 5 discusses the 
potential consequences and limitations of these 
(opened-ended) criteria. 



The methodology used to evaluate these issues is the 
topic of Chapter 6. It discusses the need to create a 
special modified universe file and describes some of 
the difficulties inherent in comparing diverse secondary 
sources to the FLICS universe and final survey frame. 
It also provides specific methodological details on 
topics covered by the first five chapters. 

Section 1.1 Principal Findings 

1 . Although the 1994 FLICS was the sixth survey of 
federal libraries, its immediate predecessor was 
conducted over fifteen years earlier, denying the 
project a current and updated listing of federal 
facilities. Thus, the first task for the FLICS was 
constructing a survey frame from scratch. While 
various directories of federal libraries existed, 
none of them had the same focus or shared the 
same definitions as the FLICS. These constraints, 
as well as the time constraints inherent in such a 
project, made this task especially difficult. 
Moreover, a large portion of the resources devoted 
to the survey had to be expended on other 
methodological problems, particularly non- 
response follow-up. 

2. The first phase of the 1994 FLICS was to create a 
universe of all possible survey candidates. Nearly 
two-thirds of the original universe file was ruled 
out-of-scope. Over 38 percent of these out-of- 
scope facilities were purposely excluded from the 
final survey frame. Because of incomplete data in 
the survey files, it was difficult to pinpoint why so 
many of the remaining potential survey candidates 
fell out-of-scope. 

3. The purposely-excluded facilities consisted 
primarily of elementary and secondary schools 
operated by the Department of Defense and 
Bureau of Indian Affairs and of federal libraries 
located overseas. The assumption underlying the 
exclusion of BLA schools, viz. that they were 
canvassed in other surveys, was largely confirmed 
although some were found in school library files 
other than the Indian School one. 

4. Comparing other library directories to the FLICS 
universe resulted in a match rate of nearly 90 
percent. Of the 1,602 federal libraries found in 
these secondary sources, only 1 89 facilities could 
not be matched to the FLICS files. How many of 
these facilities belong on the final survey frame 
(and therefore represent an under-reporting factor), 
are included in other survey responses, or fail to 



meet all four screening criteria cannot be 
determined without very extensive follow-up. 

5. This comparison also revealed that 312 library 
facilities that were on the FLICS universe and 
matched to one or more secondary source, were 
not on the final survey frame. That is, these 
facilities were viewed as bona fide libraries by at 
least two independent sources, but still failed to 
meet the FLICS inclusion criteria. The reasons for 
excluding these facilities could not be determined 
because of incomplete data in the survey files. 

6. The classification of information clearinghouses 
was inconsistent. This group of information 
facilities was declared out-of-scope during the 
universe building stage, yet seventeen facilities 
with “clearinghouse” in their name appear on the 
final survey frame. 

7. The FLICS relied on respondents to determine if 
they were considered to be a federal operation. 
This procedure contrasts with standards employed 
elsewhere in the federal establishment, including 
the Office of Management and Budget, the 
National Archives, and the Census Bureau, which 
rely on organizational structure, budgetary 
authority, and administrative criteria. 

8. The FLICS definition of a “federal” library relied 
in part on the facility's level of federal funding. A 
consequence of using such a qualifier was a survey 
frame based partially on self-reporting by the 
respondent. These responses were not 
corroborated and their reliability and validity are 
unknown. 

9. Implicit in the use of a criterion that depends on 
self-reporting is the need for a valid response. 
Consequently, the lack of alternative sources and 
corroborative data means that the completeness 
and accuracy of these responses, and therefore of 
the makeup of the survey frame, are unknown. 

10. Development of a survey frame based on a 
definition that goes beyond the generally-accepted 
structural make-up, budget authority, and 
administrative characteristics criteria may result in 
coverage overlap with other NCES-sponsored 
surveys. This evaluation also identified 12 
academic libraries that appear on both the FLICS 
and academic library survey files. 



Section 1.2 Recommendations 

1. The out-of-scope facilities in the universe file 
should be re-examined to determine whether they 
were properly classified. Special attention should 
be applied to those facilities which meet any of 
these conditions: (a) were listed as a library in 
more than one source; (b) reported in Oryx’s 
library directory as having a substantial number of 
employees or volumes; or (c) failed only the third 
criterion of the four-part screening criteria (i.e., 
verify whether the facility was a federal operation, 
perhaps by referring to the established listings 
cited in recommendation 9, rather than relying 
totally on self-classification). 

2. Federal libraries found in secondary directories, 
but not matched to the FLICS universe (listed in 
Appendix B), should be followed-up to determine 
if any should be added to the survey frame. 

3. Better documentation should be maintained on 
decisions to classify facilities as in-scope or out- 
of-scope. This will reduce confusion over the 
basis for inclusion of a facility and will facilitate 
the update of the survey frame in future surveys. 

4. More detailed and meaningful codes should be 
created to indicate why a library facility was ruled 
out-of-scope. These codes should be applied in a 
consistent manner to avoid large numbers of 
facilities in the “All Other” category. 

5. The classification of information clearinghouses 
should be reviewed and clarified. If it is possible 
for some members of this group to qualify as in- 
scope, then all members should be screened to 
identify those that meet the survey’s criteria for 
inclusion. 



6. Libraries operated by regional offices of federal 
agencies should be examined more closely for 
consistent classification. Additional scrutiny is 
needed in cases where some of the regional 
libraries are in-scope and others out-of-scope. 
Also, if some of the agency’s regional offices 
operate libraries, then its other regional offices 
should be contacted to determine if they operate 
libraries as well. 

7. Use of a funding-based criterion to define 
“federal” libraries in future surveys should be 
reconsidered. Such an open-ended definition is 
not clear, subject to different interpretations, and 
creates obstacles to achieving a complete and 
accurate survey frame. 

8. In defining what constitutes the “federal universe,” 
consideration should be given to using selection 
criteria that correspond with generally-accepted 
definitions and are consistent with those applied in 
related surveys. This will increase precision and 
help to avoid overlap in coverage. 

9. One or more established listings of federal 
agencies should be used as the basis for defining 
the FLICS’ “federal” universe rather than relying 
on respondents’ interpretation. These include such 
documents as the U.S. Government Organization 
Manual, U.S. Budget Appendix, FIPS-95 listing, 
and the Census Bureau’s Dependent Agency 
Listing for the federal government. 

10. Future surveys should take advantage of relational 
database software so that basic identification 
information about the subunits included in the 
responses of headquarter libraries can be entered 
into a separate table and linked to its parent record 
by the FedLibID code. 
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